The need for examining the internal organs of patients in a non-invasive way led to the invention of several volumetric scanning modalities like magnetic resonance (MR), computed tomography (CT), and positron emission tomography (PET) imaging. The associated scanners produce large volumetric data sets of a physical property measured on a fine volumetric grid superimposed on the subject under study.
The diverse scanning modalities produce data sets that reveal diagnostic information according to different measured physical properties. Thus, each scanning modality is suitable for a particular examination. CT scanners are more suitable to visualize the boundary between anatomical parts having a different density like for example bone versus soft tissue. MR scanners are better in differentiating soft tissue because of the differentiation of tissue according to bonding with the hydrogen atom. PET scanners measure metabolic activity making them suitable to detect early stage cancer. Besides the measured physical property, the measuring resolution of the scanners differ widely, CT being high resolution, MR being medium resolution, and PET being relatively low resolution.
Sometimes the information revealed by one particular scanner is not enough for accurate diagnosis or surgical intervention. In these cases, the patient needs to be scanned by multiple scanning modalities. With these multiple scanned volume data sets for each location in the scanned part of the patient, there are now multiple measured values to be considered. The combination of the multiple measured values enables a more accurate diagnosis or more accurate planning for surgical intervention. It is common to combine PET and CT volumes to enable better treatment of cancer.
Another instance of combined usage of multiple volume data sets occurs in follow up studies for the evaluation of the evolution of a particular illness or comparing the pre- and post-operative situation. In this case volume data sets of scanning results at different instances of time from the same or different scanning modalities will be combined.
Once the multiple volume datasets are available the examiner or physician wants to visualize the multiple volumes as one fused volume. The fused volume will appear to the examiner as one scanned volume with multiple measured values per volume grid intersection. To distinguish between the multiple measured values, often they are given a different color. A fusion operation may mix the multiple colors according to a blending operator and variable fusion weights. This enables the examiner to visualize any of the multiple volumes separately or in any fused combination.
As the measurement grids of the multiple volume data sets may differ in resolution and orientation they need to be registered. A registration process will ensure that the measurement grids of the multiple volumes are scaled and rotated so that they map to the same locations in the patient. Rigid registration consists of scaling and rotation of the multiple data sets. Non rigid registration also takes into account possible deformations of the patient. When the body shape is different at each instance of scanning, non-rigid registration is required for correctly mapping the measurement grids to the same locations in the patient.
Previously volumetric data sets could only be examined in a two-dimensional (2D) fashion by scrolling through a stack of slices. Each of the slices represents an intersection image of the patient with a virtual cutting plane. In order to examine the volumetric data sets directly in 3D, volume rendering methods have been invented displaying the volume in a three-dimensional (3D) representation. Those methods include direct volume rendering (DVR), maximum intensity projection (MIP), minimum intensity projection (MinIP), average intensity projection, digital radiography reconstruction (DRR), double contrast barium enema simulation (DCBE). Those volume rendering methods enable the examiner to rotate, zoom and pan the volumetric data set in 3D.
With fused volume rendering it becomes possible to visualize multiple registered volume data sets as one fused volume in 3D. Often the examiner wants to select a particular pixel of the fused 3D volume rendered image and perform an action on the corresponding 3D position in the patient. Finding the corresponding 3D position related to a rendered pixel is called picking. Desired picking actions may include re-centering other displayed views around the picked 3D position, adding 3D annotations to the picked 3D position including measurements and markers, starting a segmentation operation with the picked 3D position as a seed point.
Given that each pixel is the result of a fusion operation between multiple registered volume rendered images makes it not obvious how to find the picked 3D position.
Until now picking on volume rendered images was limited to single volume based volume rendering. In case of multiple volumes based fused volume rendering, no methods are known in prior art to perform picking.
Several ways of volume fusion are described in literature.
Publication [1]: “A framework for fusion methods and rendering techniques of multimodal volume data” by Maria Ferre, Anna Puig and Dani Tost; Computer Animation and Virtual Worlds Archive, Vol. 15, Issue 2 (May 2004) describes various ways of applying the fusion at all stages of the fused volume rendering pipeline. The fusion is performed by a weighted average of calculated properties at the different pipeline stages. The weights can be fixed or data range dependent.
Publication [2]: “Multi-Volume Rendering for Three-dimensional Power Doppler Imaging”, Ravi Managuli, Yang Mo Yoo, Yongmin Kim, IEEE Ultrasonics Symposium, 2005 describes similar fusion techniques and distinguishes between composite and post fusion, similar to the material/shading fusion and color fusion of publication [1]. The composite fusion however does make a data dependent selection of voxels from one of the two volumes instead of weighting.
Publication [3]: “GPU-based multi-volume rendering for the visualization of functional brain image”, F. Rössler, Eduardo Tejada, Thomas Fangmeier, Thomas Ertl, Markus Knauff; SimVis 2006 (305-318) describes an interleaved slice compositing volume fusion technique suitable for execution on GPUs, similar to material/shading fusion of [1], no weighting is done.